fix(BACKEND-TENSTORRENT-HOST-FREE-FORWARD): captured decode degenerate at the first KV block boundary (#1476) - #1498

…e at the first KV block boundary (mudler#1476)

The operator gate found captured replay deterministic-degenerate (word
salad from ~generated token 30) while host-free eager stayed coherent,
reproduced at the landed SHA in clean builds. Two root causes, both
fixed:

**The RAC page_table was `[C,1]`.** The tt-metal dataflow reader
resolves `page_table_ptr[update_idx / block_size]`
(`reader_update_cache_interleaved_start_id.cpp:110-111`) — it walks the
STICK — so the moment `cur_pos` crossed `block_size` (32) the kernel
indexed past the one-column tensor and wrote KV into a garbage physical
block. The device tensor now carries the user's whole block-table row
`[C, block_table_cols]`; ANY width change (`!=`, block-boundary growth or
the multi-request shrink when the longest request finishes — the old `>`
let the else-branch copy_to_device TT_FATAL on a shape mismatch)
reallocates and RETIRES (keeps alive) the superseded tensor, because a
freed device buffer can hand its address to a new allocation while a
recorded trace still addresses it; steady state refreshes content only on
change (`ptv != e.pt_host`), so there are zero copies inside a block.
This lands the "Phase 2 full" refresh the old comment owed and never
implemented.

**`WarmDecodePos` keyed its skip on `GraphCapturesDone()`.** That
counter is process-global and `Reset()` never clears it, so the cold
eager step after a boundary reset ran no plus_one and the RE-captured
trace read `cur_pos` one position behind. The regime flag now comes
from the driver (`s.graph.captured()`); every cold/warm/capture step
re-seeds `cur_pos = seq_lens - 1`, and only true replays leave it to
the captured plus_one.

| Check | Result |
|---|---|
| captured vs host-free eager, 80 tokens | **NOT byte-identical** — 284B md5 `3b5a579d…` vs 286B md5 `f5ffdf6a…`, first diff byte 174, both arms coherent. Adjudicated per-step top-2 (`VT_TT_DUMP_KV`): 45/80 steps argmax-identical with top-2 values agreeing to ≤0.5 logits; the FIRST divergence (step 46) is a swapped top-2 near-tie — gaps 0.25 vs 0.125 nats (1-2 bf16 ULP), the band this repo tracks for Qwen3-0.6B on TT (mudler#1488 owes the teacher-forced re-adjudication); the 34 later differences are prefix divergence, not numeric evidence |
| captured, 160 tokens | coherent, the 80-token answer a strict byte-prefix, 5 boundaries crossed, exit 0 |
| captured + `VT_TT_RECAPTURE_EVERY=8` | 9 captures / 71 replays (8 mid-generation re-captures): byte-identical to the plain captured arm, same single step-46 near-tie vs eager. Without this arm the fixed-width gate never fires a `Reset()` (engine preallocates bt_cols=256), so the re-seed guarantee was previously undetected |
| mutation reds (/tmp clone, same build config) | G2 (`GraphCapturesDone()>0` early-return restored): steps 1-10 agree, RED at step 11 — non-tie divergence (argmax 220 gap 1.125 vs eager 13), text degrades. G1 (page_table `[C,1]`, no steady refresh): 32 steps agree, RED at step 33 (first step past block_size) with the word salad. Both restores sha256-verified, rebuilt, rerun green with answers byte-identical to the gate runs |
| `test_tenstorrent_backend` | 23/23 cases, 831/831 assertions, with AND without an ambient `VT_TT_HOST_FREE_DECODE` |
| `test_qwen3_paged_engine` | unchanged from pre-fix (default path; stale golden mudler#1488) |

An earlier draft of this commit claimed captured==eager byte-identical
over 80 tokens; that was a `grep -m1 '^ Answer'` first-line artifact and
is corrected here in the message, the spec, STATUS, and the mudler#1476 index
row.

- The RAC width-realloc condition is `!=` (shrink included), matching
  the driver's `cols_changed !=` reset.
- `CopyDeviceDeviceIfCapture` / `MemsetDeviceIfCapture` read
  `VT_TT_HOST_FREE_DECODE` live instead of caching it in a static: the
  inertness-guard case unsets the env mid-process and must observe the
  decline, and a suite run under an ambient flag must not pin armed
  behavior for cases that unset it. The kRopeNeox bit-exact case now
  unsets the flag itself (mirroring the inertness case), so the suite is
  green under an ambient flag (was 21/23).
- `WarmRacIdx` builds its host page-table tensor only on the copy path
  and the never-read `RacIdxEntry::idx_host` field/write is gone.
- `[TT-DUMP-LOGITS]` carries a top-2 (id, value, gap) suffix under
  `VT_TT_DUMP_KV` — the near-tie adjudication instrument.

- `test_tenstorrent_backend` exits 139 AFTER a green doctest summary
  (static `optional<ttnn::Tensor>` destruction after device close).
  Proven pre-existing by A/B stash/build/run at `origin/main`: mudler#1486.
- `test_release_metadata` reds on every aarch64 host (fixture stages
  the host `/bin/true` into an x86_64-named archive), so preflight
  cannot go green on the TT dev fleet: mudler#1487. It is the one gate that
  stays red in this change's preflight.
- The TT `test_qwen3_paged_engine` golden is stale (anchor drift
  prompt[1] tok=10, identical before and after this fix — that test
  runs the default path): mudler#1488 owes the `VT_DUMP_IDS` re-adjudication.

Records: spec `## Owed`/`## Now` (including the `retired_pts`
defense-in-depth reachability caveat), the STATUS cell, and the
corrected mudler#1476 issue-index row.

Closes mudler#1476.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:zai-glm-5.3 [Maki]

Main advanced 22 commits (96 files: SPEC-DFLASH2 W4/W5, BPE quadratic
merge, MUSIC3 pricing, ROCM block-size contract, CI red repairs).
`.agents/benchmark-record.md` conflicted because both sides appended
2026-08-20 entries: resolved by union, keeping this branch's TT host-free
mudler#1476 entry and main's MUSIC3-E2E entry. `.agents/issue-index.md` merged
by union with no duplicate row (518 rows after). docs/STATUS.md and
docs/BENCHMARKS.md auto-merged; this branch's TT rows are untouched by
main's edits.

Card gates after the merge (Blackhole P150): test_tenstorrent_backend
23/23 + 831/831, and test_qwen3_paged_engine shows the identical
documented mudler#1488 anchor drift (prompt[1] tok=10 engine=14126) with no new
drift — the golden refresh rides the stacked mudler#1514.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:zai-glm-5.3 [Maki]

Brings the branch to current main so the record files resolve locally against
the current keyed rows.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Claude:claude-opus-5 [Claude Code]